C# binding into STFT pre-processing #1843

nenad1002 · 2025-10-27T20:00:12Z

No description provided.

src/ort_genai_c.cpp

src/csharp/SignalProcessor.cs

…nxruntime-genai into nebanfic/stft-genai

kunal-vaishnavi · 2025-11-07T03:14:44Z

src/csharp/SignalProcessor.cs

+namespace Microsoft.ML.OnnxRuntimeGenAI
+{
+    public static class SignalProcessor
+    {


Can these APIs be added to the MultiModalProcessor instead of introducing a new class? All multi-modal models (e.g. Whisper, Phi-4 mm, etc) use that processor for pre-processing.

kunal-vaishnavi · 2025-11-07T03:15:43Z

src/csharp/NativeMethods.cs


+        [DllImport(NativeLib.DllName, CallingConvention = CallingConvention.Winapi)]
+        public static extern int /* extError_t */ OgaSplitSignalSegments(
+            IntPtr /* const OgaTensor* */ input,


Can we follow the same tab spacings as the other native method APIs listed in this file for the new APIs?

kunal-vaishnavi · 2025-11-07T03:29:12Z

src/models/processor.cpp

 }

+template <typename T>
+OrtxTensor* MakeOrtxTensor(Generators::Tensor* src) {


Since the Generators namespace is included at the top, I think you can remove the Generators:: part before each Tensor.

onnxruntime-genai/src/models/processor.cpp

Line 7 in a12c530

namespace Generators {

kunal-vaishnavi · 2025-11-07T03:42:09Z

src/ort_genai_c.cpp

+    const OgaTensor* hop_ms_tensor,
+    const OgaTensor* energy_threshold_db_tensor,
+    OgaTensor* output0) {
+  OGA_TRY


Can we add unit tests for the new APIs in ORT GenAI's C API tests?

kunal-vaishnavi · 2025-11-07T03:43:18Z

src/ort_genai_c.h

 */
 OGA_EXPORT OgaResult* OGA_API_CALL OgaProcessorProcessImagesAndAudiosAndPrompts(const OgaMultiModalProcessor*, const OgaStringArray* prompts, const OgaImages* images, const OgaAudios* audios, OgaNamedTensors** input_tensors);

+OGA_EXPORT OgaResult* OGA_API_CALL OgaSplitSignalSegments(


Can you add C# API tests that call these C APIs?

src/models/processor.cpp

kunal-vaishnavi · 2025-11-08T22:57:08Z

src/ort_genai_c.h

+    const OgaTensor* energy_threshold_db_tensor,
+    OgaTensor* output0);
+
+OGA_EXPORT OgaResult* OGA_API_CALL OgaMergeSignalSegments(


Let's add comments above these new APIs to explain their usage and their parameters.

kunal-vaishnavi · 2025-11-08T22:57:12Z

src/ort_genai_c.h

+
+OGA_EXPORT OgaResult* OGA_API_CALL OgaMergeSignalSegments(
+    const OgaTensor* segments_tensor,
+    const OgaTensor* merge_gap_ms_tensor,


Given that the type of the object is an OgaTensor*, I think we can remove the _tensor suffix from the parameter names.

kunal-vaishnavi · 2025-11-08T22:57:23Z

src/ort_genai_c.cpp

+      reinterpret_cast<const Generators::Tensor*>(input),
+      reinterpret_cast<const Generators::Tensor*>(sr_tensor),
+      reinterpret_cast<const Generators::Tensor*>(frame_ms_tensor),
+      reinterpret_cast<const Generators::Tensor*>(hop_ms_tensor),


To set the return value to output0 in a way that does not cause memory corruptions, you may need to use ReturnShared<OgaTensor> and change output0 to OgaTensor**.

onnxruntime-genai/src/ort_genai_c.cpp

Lines 632 to 638 in cac5438

OgaResult* OGA_API_CALL OgaTokenizerEncodeBatch(const OgaTokenizer* tokenizer, const char** strings, size_t count, OgaTensor** out) {

OGA_TRY

auto tensor = tokenizer->EncodeBatch(std::span<const char*>(strings, count));

*out = ReturnShared<OgaTensor>(tensor);

return nullptr;

OGA_CATCH

}

Let's also rename output0 to out since there's just one output and to be consistent with existing API conventions.

It doesn't cause memory corruption anymore, code I have here now is working correctly, so I wouldn't mess up again with changing it.

kunal-vaishnavi · 2025-11-08T22:57:37Z

src/ort_genai_c.cpp

+    OgaTensor* output0) {
+  OGA_TRY
+  return reinterpret_cast<OgaResult*>(
+      Generators::MergeSignalSegments(


We can move SplitSignalSegments and MergeSignalSegments inside a processor object. The outputs from those methods can then be set to the output tensors from the processor APIs.

auto tensor = processor->SplitSignalSegments(...); *out = ReturnShared<OgaTensor>(tensor);

auto tensor = processor->MergeSignalSegments(...); *out = ReturnShared<OgaTensor>(tensor);

Then these new APIs could be called similar to how methods such as OgaTokenizerEncodeBatch look in the above review comment. This will also help add future support for these new APIs in other language bindings.

Why would I have to construct a new object which has no connection with my current functionality? I think what we need to do here is to clarify, do we even need a Processor class, and if we do, what exactly should be inside it, because I see many methods not being a part of the class (e.g. ProcessTensor is not a part of the class but is within the same file).

I can do this, but we really need to clarify this.

kunal-vaishnavi · 2025-11-08T22:57:58Z

src/csharp/SignalProcessor.cs

+            {
+                long[] inputShape = new long[] { 1, inputSignal.Length };
+
+                input = CreateFloatTensorFromArray(inputSignal, inputShape);


Can you construct a Tensor object directly for the input data? Each Tensor object has its own IntPtr handle that you can provide for the NativeMethods.CAPI call.

onnxruntime-genai/src/csharp/Tensor.cs

Lines 28 to 45 in 9d3631d

public class Tensor : IDisposable

{

private IntPtr _tensorHandle;

private bool _disposed = false;

public Tensor(IntPtr data, Int64[] shape, ElementType type)

{

Result.VerifySuccess(NativeMethods.OgaCreateTensorFromBuffer(data, shape, (UIntPtr)shape.Length, type, out _tensorHandle));

}

internal Tensor(IntPtr tensorHandle)

{

Debug.Assert(tensorHandle != IntPtr.Zero);

_tensorHandle = tensorHandle;

_disposed = false;

}

internal IntPtr Handle { get { return _tensorHandle; } }

The Tensor class can help with memory management during disposal.

Here is an example with Tensor from a unit test.

onnxruntime-genai/test/csharp/TestOnnxRuntimeGenAIAPI.cs

Lines 637 to 669 in 9d3631d

public void TestTensorAndAddExtraInput()

{

string modelPath = _tinyRandomGpt2ModelPath;

using var model = new Model(modelPath);

Assert.NotNull(model);

using var generatorParams = new GeneratorParams(model);

Assert.NotNull(generatorParams);

float[] data = { 0, 1, 2, 3, 4, 10, 11, 12, 13, 14, 20, 21, 22, 23, 24 };

long[] shape = { 3, 5 };

// Pin the array to get its pointer

GCHandle handle = GCHandle.Alloc(data, GCHandleType.Pinned);

try

{

IntPtr data_pointer = handle.AddrOfPinnedObject();

using var tensor = new Tensor(data_pointer, shape, ElementType.float32);

Assert.NotNull(tensor);

Assert.Equal(shape, tensor.Shape());

Assert.Equal(ElementType.float32, tensor.Type());

using var generator = new Generator(model, generatorParams);

Assert.NotNull(generator);

generator.SetModelInput("test_input", tensor);

}

finally

{

handle.Free();

}

}

Yes, this approach gave me memory corruption as well, there might be an issue with Tensor class, but I can give it another try.

kunal-vaishnavi · 2025-11-08T22:58:51Z

src/models/processor.cpp

+  const OrtxTensor* frame_tensor = Generators::MakeOrtxTensorConst<int64_t>(frame_ms_tensor);
+  const OrtxTensor* hop_tensor = Generators::MakeOrtxTensorConst<int64_t>(hop_ms_tensor);
+  const OrtxTensor* thr_tensor = Generators::MakeOrtxTensorConst<float>(energy_threshold_db_tensor);
+  OrtxTensor* out_tensor = Generators::MakeOrtxTensor<int64_t>(output0);


I would recommend looking at how the tokenizer and processor APIs are currently implemented (ex: Process method in the Whisper processor). That will help with making the implementations of the two methods similar to existing implementations.

You can use CheckResult in place of the err check.

Are tensors really needed for most of the parameters passed to OrtxSplitSignalSegments? The sampling rate (sr_tensor), frame size (frame_ms), hop length (hop_ms), and energy threshold (energy_threshold_db) are singular values. Can we modify OrtxSplitSignalSegments to use primitive types so that primitive types can be used in ORT GenAI for the "language binding API --> ORT GenAI C API --> ORT GenAI C++ API" flow? That will also greatly clean up the need for all of the workarounds you have in the C# bindings and in this file.

If OrtxSplitSignalSegments is updated, you can have the output stored as OrtxTensorResult** result inside that extensions API. Then you can create the output tensor inside this method instead of passing it in as a parameter to SplitSignalSegments. Here is an example.

onnxruntime-genai/src/models/whisper_processor.cpp

Lines 27 to 31 in 9d3631d

ort_extensions::OrtxObjectPtr<OrtxTensorResult> result;

CheckResult(OrtxFeatureExtraction(processor_.get(), audios->audios_.get(), result.ToBeAssigned()));

ort_extensions::OrtxObjectPtr<OrtxTensor> mel;

CheckResult(OrtxTensorResultGetAt(result.get(), 0, mel.ToBeAssigned()));

Its more consistant having tensors, but I can do your approach as well, I have no preference, I was just trying to be consistant + I haven't seen built-in types used for Oga API.

It is possible, yes, but since I am initializing tensors on C# side, again for consistancy and simplicity (no need for **), I decided to create all on the caller side. But the extension should, in theory, work for both cases.

nenad1002 · 2025-11-10T17:11:09Z

Thanks for reviewing the PR @kunal-vaishnavi , I am currently still testing the changes from here with Audio Streaming, and will for now only address the comments that are fast to resolve, while others will follow later.

nenad1002 and others added 7 commits October 24, 2025 17:46

Stft genai

99687be

Merge method

4715a9a

Clean model.cpp

cdb6325

Clean c# code

7f3a634

Some changes

dcb2c14

Make sure to return segments arr with correct size

835628a

Merge branch 'main' into nebanfic/stft-genai

fbe74ff

nenad1002 requested a review from kunal-vaishnavi October 28, 2025 01:45

kunal-vaishnavi reviewed Oct 30, 2025

View reviewed changes

src/ort_genai_c.cpp Outdated Show resolved Hide resolved

kunal-vaishnavi reviewed Oct 30, 2025

View reviewed changes

src/ort_genai_c.cpp Outdated Show resolved Hide resolved

kunal-vaishnavi reviewed Oct 30, 2025

View reviewed changes

src/csharp/SignalProcessor.cs Outdated Show resolved Hide resolved

nenad1002 added 8 commits October 30, 2025 15:23

Resolve cpp comm

febc439

Merge branch 'nebanfic/stft-genai' of https://github.com/microsoft/on…

ad4de9b

…nxruntime-genai into nebanfic/stft-genai

Remove static link

595f96c

Return include back

a2f4773

Verify CUDA weak linkage problem

80b9430

Generators::

9280016

Remove 'fake' template inst

bf29508

Move ortx logic to processor

a12c530

nenad1002 requested a review from kunal-vaishnavi November 3, 2025 21:26

kunal-vaishnavi reviewed Nov 7, 2025

View reviewed changes

sayanshaw24 reviewed Nov 7, 2025

View reviewed changes

src/models/processor.cpp Show resolved Hide resolved

nenad1002 added 2 commits November 7, 2025 22:56

Try pinnig nmemory

7a9527d

Update comments

cac5438

kunal-vaishnavi reviewed Nov 8, 2025

View reviewed changes

	OgaResult* OGA_API_CALL OgaTokenizerEncodeBatch(const OgaTokenizer* tokenizer, const char strings, size_t count, OgaTensor out) {
	OGA_TRY
	auto tensor = tokenizer->EncodeBatch(std::span<const char*>(strings, count));
	*out = ReturnShared<OgaTensor>(tensor);
	return nullptr;
	OGA_CATCH
	}

	public class Tensor : IDisposable
	{
	private IntPtr _tensorHandle;
	private bool _disposed = false;

	public Tensor(IntPtr data, Int64[] shape, ElementType type)
	{
	Result.VerifySuccess(NativeMethods.OgaCreateTensorFromBuffer(data, shape, (UIntPtr)shape.Length, type, out _tensorHandle));
	}

	internal Tensor(IntPtr tensorHandle)
	{
	Debug.Assert(tensorHandle != IntPtr.Zero);
	_tensorHandle = tensorHandle;
	_disposed = false;
	}

	internal IntPtr Handle { get { return _tensorHandle; } }

	public void TestTensorAndAddExtraInput()
	{
	string modelPath = _tinyRandomGpt2ModelPath;
	using var model = new Model(modelPath);
	Assert.NotNull(model);

	using var generatorParams = new GeneratorParams(model);
	Assert.NotNull(generatorParams);

	float[] data = { 0, 1, 2, 3, 4, 10, 11, 12, 13, 14, 20, 21, 22, 23, 24 };
	long[] shape = { 3, 5 };

	// Pin the array to get its pointer
	GCHandle handle = GCHandle.Alloc(data, GCHandleType.Pinned);
	try
	{
	IntPtr data_pointer = handle.AddrOfPinnedObject();

	using var tensor = new Tensor(data_pointer, shape, ElementType.float32);
	Assert.NotNull(tensor);

	Assert.Equal(shape, tensor.Shape());
	Assert.Equal(ElementType.float32, tensor.Type());

	using var generator = new Generator(model, generatorParams);
	Assert.NotNull(generator);
	generator.SetModelInput("test_input", tensor);
	}
	finally
	{
	handle.Free();
	}
	}

	ort_extensions::OrtxObjectPtr<OrtxTensorResult> result;
	CheckResult(OrtxFeatureExtraction(processor_.get(), audios->audios_.get(), result.ToBeAssigned()));

	ort_extensions::OrtxObjectPtr<OrtxTensor> mel;
	CheckResult(OrtxTensorResultGetAt(result.get(), 0, mel.ToBeAssigned()));

C# binding into STFT pre-processing #1843

Are you sure you want to change the base?

C# binding into STFT pre-processing #1843

Uh oh!

Conversation

nenad1002 commented Oct 27, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kunal-vaishnavi Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nenad1002 Nov 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nenad1002 Nov 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nenad1002 commented Nov 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kunal-vaishnavi Nov 7, 2025 •

edited

Loading

nenad1002 Nov 10, 2025 •

edited

Loading

nenad1002 Nov 10, 2025 •

edited

Loading